The rapid growth of networks in information systems has resulted in the increasing attention to IDS (Intrusion Detection Systems)—such as NIDS (Network Intrusion Detection Systems), HIDS (Host Intrusion detection system)- and NIPS (Network Intrusion Prevention Systems), NIPS combining both firewall and NIDS.
Computer networks have to be protected against DoS (Denial of Service) attacks, unauthorized disclosure or manipulation of information and the modification or destruction of data. In the meantime, the availability, confidentiality and integrity of critical information systems have to be provided.
It is reported that there have been 10,000 new viruses or variants of existing viruses in the year 2004 and at least one new attack spotted every hour (Kay, “Low volume viruses: new tools for criminals”, Network Secur. 6, 2005, pp. 16-18). In 2001, the Code Red worm propagated to over 359,000 Internet hosts in less than 14 hours (Moore et al, “Code Red: a case study on the spread and victims of an internet worm”, Proceeding of the second ACM Internet measurement workshop, 2002). In 2003, the SQL Slammer worm propagates to over 75,000 hosts in less than 30 minutes, 90% of which were infected within 10 minutes (Moore et al, “The spread of the sapphire/slammer worm technical report” CAIDA technical report, 2003). In 2002, a US federal bureau of investigation survey reported that the average cost of a successful attack by a external hacker is 56,000 USD, the average cost of a successful insider attack being reported at 2.7 million USD. (Power “2002 CSI/FBI computer crime and security survey” Computer security issues and trends, vol VIII, no1, spring 2002).
IDS conventionally designate some software with the functions of detecting, identifying and responding to unauthorized or abnormal activities on a target system.
IDS have traditionally been centralized in design, centralized IDS being typically installed at a choke point of the network, e.g. the network service provider gateway, and operating in a standalone mode with centralized applications physically integrated within a single processing unit. Distributed IDS also exists, consisting of multiple sensors deployed on different areas of a large network, all of which ultimately report to a central server that aggregates the information and processes it.
The purpose of IDS is to distinguish between intruders and normal users. The goal of the IDS is to provide a mechanism for the detection of security violations either in real time or batch mode. Violations are initiated either by outsiders attempting to break into a system, or by insiders attempting to misuse their privileges.
The major functions performed by IDS are: monitoring and analyzing user and system activities, assessing the integrity of critical system or data files, recognizing activity patterns reflecting known attacks, responding automatically to detected activity and reporting the outcome of the detection process.
Intrusion detection can be divided into three categories based on the detection method: misuse detection, anomaly detection and data mining. Hybrid intrusion methods are also known, combining two approaches at the same time. Known as KDD-99, a labeled data set for comparing detection methods has been provided by the International Knowledge Discovery and Data Mining Tools Competition.
Misuse detection works on searching for the traces or patterns of well known attacks. Misuse detection systems try to match computer activity to stored signatures of known exploits or attacks. It uses a priori knowledge on attacks to look for attack traces. In other words, misuse detection refers to techniques that use patterns of known intrusions or weak spots of a system (e.g. system utilities that have the buffer overflow vulnerabilities) to match and identify intrusion.
The sequence of attack actions, the conditions that compromise a system's security, as well as the evidence (e.g. damage or system logs) left behind by intrusions can be represented by a number of general patterns matching models. These pattern matching models encode known signature as patterns that are then matched against audit data. Pattern matching frequently refers to fuzzy logic and artificial intelligence techniques such as neural networks.
For example NIDES (Next generation Intrusion Detection Expert System) uses rules to describe attack actions, STAT (State Transition Analysis Tool) uses state transition diagrams to model general states of the system and access control violations, and IDIOT (Intrusion Detection In Our Time) uses colored nets to represent intrusion signatures as sequences of events on the target system.
The key advantage of misuse detection systems is that once the patterns of known intrusions are stored, future instances of these intrusions can be detected efficiently and effectively.
However, newly invented attacks will likely go undetected, leading to unacceptable false negative error rates. Although misuse detection is assumed to be more accurate than anomaly detection, the major drawback of this technique is in creating a signature that encompasses most possible variations of intrusive and non intrusive activities.
Anomaly detection uses a model of the normal user or system behavior (user and system profile) and flags significant deviations from this model as potentially malicious. For example, the CPU usage and the frequency of system commands during a user login session are statistical parameters included in the user's profile. Deviation from a profile can be computed as the weight sum of the deviations of the constituent statistic.
The key advantage of anomaly detection systems is that they can detect unknown intrusion since they require non a priori knowledge about specific intrusion.
However, defining and maintaining normal profile is a nontrivial and error prone task, leading to sometimes unacceptable levels of false alarms.
Many recent approaches to IDS have utilized data mining techniques, e.g. CMDS (Computer Misuse Detection System), IDES (Intrusion Detection Expert System), MIDAS (Multics Intrusion Detection and Alerting System).
Data mining based IDS collect data from sensors such as e.g. those available from Cyber-Patrol Inc. Sensors monitor some aspect of a system, such as network activity, system calls used by user processes, and file system accesses. They extract predictive features from the raw data stream being monitored to produce formatted data that can be used for detection.
For a network based attack system, JAM uses frequent episode mining that generates the normal usage patterns of a specific node in the network. These patterns are used to build a base classifier that determines the abnormality of the network node. In order to guarantee correct classification, a sufficient amount of normal and abnormal data should be gathered for the learning phase of a classifier. A set of base classifiers can be used to build a meta classifier, since each base classifier monitors a different node of a network, an intrusion of the network can be detected by meta classifiers combining the results of its base classifier.
IDS are categorized according to the kind of audit source location they analyze.
Most IDS are classified as either a network based intrusion detection or a host based intrusion detection approach for recognizing and deflecting attacks.
When IDS look for these patterns in the network traffic, they are classified as network based intrusion detection.
Network based IDS analyze network packets that are captured on a network. As an example, SNORT is an open source network intrusion detection system, capable of performing real-time traffic analysis and packet logging on IP networks. SNORT can not generate intrusion patterns automatically. Experts must first analyze and categorize attack packets and hand code the corresponding patterns and rules for misuse detection. The number of patterns is increasing and is than 2100 in the current SNORT release.
When IDS look for attack signatures in the log files, they are classified as host based intrusion detection. Host based IDS system are installed locally on host machines. Host based IDS analyze host bound audit sources such as operating system audit trails, system logs, and application logs. In other words, host based IDS systems evaluate the activities and access to key servers upon which a Host based IDS has been placed.
The current IDS have contributed to identify attacks using historical patterns. But they have difficulty in identifying attacks using a new pattern or with no pattern. Using a rule based approach such as USTAT (State Transition Analysis Tool for Unix), NADIR (Network Anomaly Detection and Intrusion Reporter), and W&S (Wisdom and Sense), slight variations in a an attack sequence can affect the activity rule comparison to a degree that intrusion is not detected by the intrusion detection mechanism.
Two types of errors result in evoking inevitable IDS costs. These errors consist of false positive and false negative errors in IDS.
The false positive errors occur because the IDS sensor misinterprets normal packets or activities as an attack. False negative errors occur because an attacker is misclassified as a normal user.
It has been estimated that up to 99% of alerts reported by IDSs are not related to security issues (Julish “Using root cause analysis to handle intrusion detection alarm, PhD thesis University of Dortmund, 2003, page 1). Reasons for this include the following. Firstly, in many cases an intrusion differs only slightly from normal activities. Owing to harsh real time requirements, IDSs cannot analyze the context of all activities to the required extend. Secondly, writing signature for IDSs is a very difficult task. In some cases, it can be difficult to determine the right balance between an overly specific signature (which is not able to capture all attacks or their variations) and an overly general one (which recognizes legitimate actions as intrusions). Thirdly, actions that are normal in certain environments may be malicious in others. Fourthly, assuming that one million packets containing twenty packets of intrusions are analyzed per day, a perfect detection rate of 1.0 and a very low false positive rate of the order of 10−5 is leading to ten false positives, i.e. a Bayesian detection rate of true positive of only 66%.
These false positive intrusion alerts are a crucial issue that curbs the evaluation and resolution of real intrusion incidents. This amount of false-positive has an important negative effect on any correlation process that follows intrusion alerts, whatever it is automatic or human-being based. Indeed, state of the art studies on intrusion detection have demonstrated that a huge amount of false-positive alert decreases drastically the performance of automatic correlation engines that try to link several alerts to detect multi-step complex attacks (Ning et al “Learning attack strategies from intrusion alert” ACM conference on computer and communications security, 2003).
If the correlation is performed by a human expert, a huge amount of false-positive alerts tends to distract him as he tries to detect dangerous attacks. This makes the finding of real intrusion more difficult.
To give a real world example, up to 10 Gigabytes of security logging can be generated daily by roughly 15 sensors. After correlation, about one hundred of alerts are transmitted daily to the security management system and after analysis by a human expert, only a ten of cases per day are considered as “look like dangerous”.
Various solutions have been proposed to address the issue of intrusion alert false-positive reduction.
Most of those solutions relate to alert correlation. Correlation techniques can be classified in several categories. First, correlation means bring together several intrusion alerts relating to the same dangerous event (i.e. the same attack). A second means is bringing together several intrusion alerts relating to several dangerous events in order to determine if a complex attack is ongoing within the network.
Although instigators of alert correlation had first expected it could reduce the amount of false-positive alerts, it is now known that false-positive alerts curb the performance of common correlation engines (Ning et al “Learning attack strategies from intrusion alert” ACM conference on computer and communications security, 2003). Moreover, the correlation is already a computer time-consuming task that exposes a correlation system to DoS attacks by false-positive flooding.
Another approach to reduce the false-positive alerts consists in using contextual information on the infrastructure (e.g. network topology, known existing vulnerabilities) to determine if the attack has some chances to be successful and figures a real intrusion possibility. This technique relates to the concept of alert verification. In literature, two kinds of alert verification exist, active and passive.
Active alert verification uses information gathered after an alert has been raised to determine if the attack is successful, while passive verification uses a priori information of the infrastructure security to determine if the attack has a chance to be successful.
Current passive verification systems use static knowledge of the infrastructure security and do not measure it. This can lead to misclassification of alerts as false-positive and thus, creates false-negatives (alert is not generated for a real attack and is classified as false-positive).
On the other side, current active alert verification systems are based on a posteriori (after the intrusion alert has been issued) gathering of information that may prove the success of the attack (i.e. a signature of the intrusion, compared to an attack signature commonly used by IDS/IPS). In that case, the main issue is that the verification could occur after the attacker has covered the track of its intrusion.